{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a5440062-acd7-4ede-b544-904384a4748a",
   "metadata": {},
   "source": [
    "# LangSmith Tracing 快速入门\n",
    "\n",
    "为了使用追踪（Tracing）功能，必须将`LANGCHAIN_TRACING_V2`环境变量设置为`true`，以便在使用 @traceable或traceable时将跟踪日志记录到LangSmith。使得开发者可以在不更改代码的情况下切换跟踪开关。\n",
    "\n",
    "此外，还需要将 `LANGCHAIN_API_KEY` 环境变量设置好（注册和创建`API_KEY`官方文档链接：https://docs.smith.langchain.com/）\n",
    "\n",
    "### 追踪记录方法\n",
    "\n",
    "本教程将展示多种使用 LangSmith 来记录追踪 LLM 生成内容和性能的方法：\n",
    "- 使用 `@traceable` 装饰器追踪特定 Python 函数\n",
    "- 使用 `wrap_openai` 方法自动追踪 OpenAI 客户端所有调用\n",
    "- 使用 `RunTree` API\n",
    "- 使用 `trace` 上下文管理\n",
    "- 结合 `LangChain` 来追踪记录\n",
    "\n",
    "### 记录到特定项目\n",
    "\n",
    "追踪日志默认会记录在 `default` 项目中，如果想要记录在其他项目，需要在 LangSmith 平台新建项目。然后使用以下方式：\n",
    "- 使用环境变量全量修改：`export LANGCHAIN_PROJECT=my-custom-project`\n",
    "- 动态修改单条记录：`@traceable(project_name=\"my-custom-project\")`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b98a0e57-0ba6-4e98-9485-22fdca6bd694",
   "metadata": {},
   "source": [
    "## @traceable 装饰器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5956cb98-b567-4a2f-9ead-1b14742e8bc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import traceable\n",
    "from openai import Client\n",
    "\n",
    "# 创建 OpenAI 客户端\n",
    "openai = Client()\n",
    "\n",
    "# 标记函数可追踪\n",
    "@traceable\n",
    "def format_prompt(subject):\n",
    "    # 格式化提示信息\n",
    "    return [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"你是一个乐于助人的助手。\",\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": f\"对于一家卖{subject}的店来说，取个什么名字好呢？\"\n",
    "        }\n",
    "    ]\n",
    "\n",
    "# 标记函数可追踪，并指定运行类型为 LLM\n",
    "@traceable(run_type=\"llm\")\n",
    "def invoke_llm(messages):\n",
    "    # 调用 OpenAI 的聊天模型\n",
    "    return openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo\", temperature=0\n",
    "    )\n",
    "\n",
    "# 标记函数可追踪\n",
    "@traceable\n",
    "def parse_output(response):\n",
    "    # 解析并返回模型的输出\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "# 标记函数可追踪\n",
    "@traceable\n",
    "def run_pipeline(prompt):\n",
    "    # 运行整个管道流程\n",
    "    messages = for mat_prompt(prompt)  # 创建提示信息\n",
    "    response = invoke_llm(messages)  # 调用模型\n",
    "    return parse_output(response)  # 解析模型输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "834711de-9f4a-40e5-90ef-d4fd16c2faff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'为一家卖烤鸭的店取名字时，可以考虑以下一些建议：\\n\\n1. 金陵烤鸭坊\\n2. 鸭香阁\\n3. 鸭乐园\\n4. 鸭舫\\n5. 鸭悦坊\\n6. 鸭乐食府\\n7. 鸭香居\\n8. 鸭乐轩\\n9. 鸭乐园\\n10. 鸭乐坊\\n\\n希望这些建议能够帮助你取一个好听且有吸引力的店名！'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_pipeline(\"烤鸭\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c2a5528b-9051-4b28-ac04-90579471650f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'为一家卖驴肉火烧的店取名可以考虑以下几个方向：\\n1. 与驴肉相关的名字：比如“驴肉香坊”、“驴肉乡村”等。\\n2. 引人入胃的名字：比如“香味驴肉坊”、“美味驴肉馆”等。\\n3. 独特创意的名字：比如“驴肉烧烤屋”、“驴肉烧的香”等。\\n\\n希望以上建议能够帮助你取一个好听的店名！'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_pipeline(\"驴肉火烧\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e0ebb94-1e99-4ca4-95c2-1837eb326b20",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c0c03173-5ead-4f9e-b6ed-d97cf21c0f56",
   "metadata": {},
   "source": [
    "## wrap_openai 客户端\n",
    "\n",
    "Python/TypeScript 中的 wrap_openai/wrapOpenAI 方法允许您包装 OpenAI 客户端，以便自动记录跟踪 - 无需装饰器或函数包装！\n",
    "\n",
    "该包装器与 @traceable 装饰器或可追溯函数完美配合，可以在同一应用程序中同时使用两者。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b3c43988-c21a-4de7-95a6-5c7d8c877d7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "from langsmith import traceable\n",
    "from langsmith.wrappers import wrap_openai\n",
    "\n",
    "# 包装 OpenAI 客户端\n",
    "client = wrap_openai(openai.Client())\n",
    "\n",
    "# 标记函数可追踪，并指定运行类型为工具，名称为\"Retrieve Context\"\n",
    "@traceable(run_type=\"tool\", name=\"Retrieve Context\")\n",
    "def my_tool(question: str) -> str:\n",
    "    # 返回上下文信息\n",
    "    return \"在今天早上的会议中，我们讨论了出海创业的机会与挑战。\"\n",
    "\n",
    "# 标记函数可追踪，名称为\"Chat Pipeline\"\n",
    "@traceable(name=\"Chat Pipeline\")\n",
    "def chat_pipeline(question: str):\n",
    "    # 获取上下文信息\n",
    "    context = my_tool(question)\n",
    "    \n",
    "    # 创建消息列表，包含系统消息和用户消息\n",
    "    messages = [\n",
    "        { \"role\": \"system\", \"content\": \"你是一个乐于助人的助手。请仅根据给定的上下文回复用户的请求。\" },\n",
    "        { \"role\": \"user\", \"content\": f\"问题：{question}\\n上下文：{context}\"}\n",
    "    ]\n",
    "    \n",
    "    # 调用聊天模型生成回复\n",
    "    chat_completion = client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\", messages=messages\n",
    "    )\n",
    "    \n",
    "    # 返回模型的回复内容\n",
    "    return chat_completion.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f68fd0b1-42cf-41d6-80db-34ae698938c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'当然！在今天早上的会议中，我们主要讨论了出海创业的机会与挑战。'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 调用 chat_pipeline 函数\n",
    "chat_pipeline(\"你能总结一下今天早上的会议吗？\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c492c03-0c06-4954-a771-842c1e69ab93",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f69594a-ce78-460a-8f7c-234e71671af4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "93f104da-4557-4a19-ab78-ccbc87a77fc3",
   "metadata": {},
   "source": [
    "## RunTree API\n",
    "\n",
    "通过 RunTree API，是跟踪日志记录到 LangSmith 的另一种更直接的方式。\n",
    "\n",
    "该API允许对跟踪进行更多控制：可以手动创建运行和子运行以组装您的跟踪。\n",
    "\n",
    "您仍然需要设置LANGCHAIN_API_KEY，但对于**此方法不需要LANGCHAIN_TRACING_V2**。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b20cc85a-b244-46bb-9f60-ef1bc7dd6596",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "from langsmith.run_trees import RunTree\n",
    "\n",
    "# 用户输入的问题\n",
    "question = \"你能总结一下今天早上的会议吗？\"\n",
    "\n",
    "# 创建一个顶层的运行节点\n",
    "pipeline = RunTree(\n",
    "    name=\"Chat Pipeline\",\n",
    "    run_type=\"chain\",\n",
    "    inputs={\"question\": question}\n",
    ")\n",
    "\n",
    "# 在检索步骤中获取的上下文信息\n",
    "context = \"在今天早上的会议中，我们回顾了改革开放的重大成果。\"\n",
    "\n",
    "# 创建消息列表，包含系统消息和用户消息\n",
    "messages = [\n",
    "    { \"role\": \"system\", \"content\": \"你是一个乐于助人的助手。请仅根据给定的上下文回复用户的请求。\" },\n",
    "    { \"role\": \"user\", \"content\": f\"问题：{question}\\n上下文：{context}\"}\n",
    "]\n",
    "\n",
    "# 创建一个子运行节点\n",
    "child_llm_run = pipeline.create_child(\n",
    "    name=\"OpenAI Call\",\n",
    "    run_type=\"llm\",\n",
    "    inputs={\"messages\": messages},\n",
    ")\n",
    "\n",
    "# 生成回复\n",
    "client = openai.Client()\n",
    "chat_completion = client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo\", messages=messages\n",
    ")\n",
    "\n",
    "# 结束子运行节点并记录输出\n",
    "child_llm_run.end(outputs=chat_completion)\n",
    "child_llm_run.post()\n",
    "\n",
    "# 结束顶层运行节点并记录输出\n",
    "pipeline.end(outputs={\"answer\": chat_completion.choices[0].message.content})\n",
    "pipeline.post()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db9551af-b355-4e1e-806a-d4a5f3391e8e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f491ec79-ca16-43d1-bb35-104fdac70bc9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ba406496-fbbb-422f-b2d4-41333c5232c5",
   "metadata": {},
   "source": [
    "## trace 上下文管理器\n",
    "\n",
    "在Python中，使用trace上下文管理器跟踪日志记录到LangSmith。在以下情况下非常有用：\n",
    "\n",
    "- 想要为特定代码块记录跟踪日志，而不设置一个会为整个应用程序记录跟踪的环境变量。\n",
    "- 希望对跟踪的输入、输出和其他属性进行控制。\n",
    "- 使用装饰器或包装器并不可行。\n",
    "\n",
    "该上下文管理器与可追溯的装饰器和 `wrap_openai` 包装器无缝集成，因此，可以在同一应用程序中同时使用它们。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "21255c23-f9e4-4c96-b729-475cd8ae62f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "from langsmith import trace\n",
    "from langsmith import traceable\n",
    "from langsmith.wrappers import wrap_openai\n",
    "\n",
    "# 包装 OpenAI 客户端\n",
    "client = wrap_openai(openai.Client())\n",
    "\n",
    "# 标记函数可追踪，并指定运行类型为工具，名称为\"Retrieve Context\"\n",
    "@traceable(run_type=\"tool\", name=\"Retrieve Context\")\n",
    "def my_tool(question: str) -> str:\n",
    "    # 返回上下文信息\n",
    "    return \"During this morning's meeting, we solved all world conflict.\"\n",
    "\n",
    "def chat_pipeline(question: str):\n",
    "    # 获取上下文信息\n",
    "    context = my_tool(question)\n",
    "    \n",
    "    # 创建消息列表，包含系统消息和用户消息\n",
    "    messages = [\n",
    "        { \"role\": \"system\", \"content\": \"你是一个乐于助人的助手。请仅根据给定的上下文回复用户的请求。\" },\n",
    "        { \"role\": \"user\", \"content\": f\"问题：{question}\\n上下文：{context}\"}\n",
    "    ]\n",
    "    \n",
    "    # 调用聊天模型生成回复\n",
    "    chat_completion = client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\", messages=messages\n",
    "    )\n",
    "    \n",
    "    # 返回模型的回复内容\n",
    "    return chat_completion.choices[0].message.content\n",
    "\n",
    "# 应用输入\n",
    "app_inputs = {\"input\": \"Can you summarize this morning's meetings?\"}\n",
    "\n",
    "# 跟踪聊天管道运行\n",
    "with trace(\"Chat Pipeline\", \"chain\", project_name=\"my_test\", inputs=app_inputs) as rt:\n",
    "    output = chat_pipeline(\"Can you summarize this morning's meetings?\")\n",
    "    rt.end(outputs={\"output\": output})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c780ee8-10d8-4162-8a89-85f9aafa2ea5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e0f0d4d9-dad9-49e6-8773-689435433923",
   "metadata": {},
   "source": [
    "## 记录多模态模型 GPT-4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "99f85113-0fe2-41c9-80e0-905fbf6d1b22",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from langsmith.wrappers import wrap_openai\n",
    "\n",
    "# 包装 OpenAI 客户端\n",
    "client = wrap_openai(OpenAI())\n",
    "\n",
    "# 调用聊天模型生成回复\n",
    "response = client.chat.completions.create(\n",
    "  model=\"gpt-4-turbo\",\n",
    "  messages=[\n",
    "    {\n",
    "      \"role\": \"user\",\n",
    "      \"content\": [\n",
    "        {\"type\": \"text\", \"text\": \"介绍下这幅图讲的什么？\"},\n",
    "        {\n",
    "          \"type\": \"image_url\",\n",
    "          \"image_url\": {\n",
    "            \"url\": \"https://p6.itc.cn/q_70/images03/20200602/0c267a0d3d814c9783659eb956969ba1.jpeg\",\n",
    "          },\n",
    "        },\n",
    "      ],\n",
    "    }\n",
    "  ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c6661f6a-8d54-4fe4-96f8-8f0d046a719b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "这幅图是一种幽默搞笑的对比图。左侧展示的是一只形如肌肉男的柴犬，被称为“16岁的我”，右侧则是一只普通的柴犬，被称为“工作后的我”。图片通过夸张的肌肉和普通的狗的形态来幽默地表达了人们对比自己年轻时充满活力和成年后工作压力导致身体和精神状态“变形”的感受。左边的大肌肉柴犬下方的文字翻译为“我可以一口气做一百个俯卧撑，一条跑足十公里，浴火重生的女人，人见人爱的大男孩”，而右边的普通柴犬下方的文字翻译为“好累啊 好想赖床 浑身疼痛 我没有病 你心有病 我命由我不由天 独步天下”。这些标签富含讽刺和幽默意味，反映了现代生活中劳累与压力的普遍现象。\n"
     ]
    }
   ],
   "source": [
    "# 打印回复的内容\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cf55450-6347-4d09-8b19-1c256709e0e8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9371b424-1510-44c5-a433-09b4822305f6",
   "metadata": {},
   "source": [
    "## 结合 LangChain 记录追踪\n",
    "\n",
    "设置好以下环境变量后，无需任何额外的代码即可追踪 LangChain 运行\n",
    "```shell\n",
    "export LANGCHAIN_TRACING_V2=true\n",
    "export LANGCHAIN_API_KEY=<your-api-key>\n",
    "\n",
    "# The below examples use the OpenAI API, so you will need\n",
    "export OPENAI_API_KEY=<your-openai-api-key>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cf7f045a-5a65-4c3d-aa45-e415a38d62ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "# 从消息中创建聊天提示模板\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"你是一个乐于助人的助手。请参考给定的上下文回复用户的请求。\"),\n",
    "    (\"user\", \"问题：{question}\\n上下文：{context}\")\n",
    "])\n",
    "\n",
    "# 使用指定的模型\n",
    "model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "# 将提示模板、模型和输出解析器链在一起\n",
    "chain = prompt | model | output_parser\n",
    "\n",
    "# 定义问题和上下文\n",
    "question = \"孙悟空到底打过几次白骨精啊？\"\n",
    "context = \"其实孙悟空三打白骨精后，又打了她一次\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "dcb18175-b9ef-40d4-b455-52be5f66028c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'孙悟空一共打过四次白骨精。'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 调用链条并传递输入\n",
    "chain.invoke({\"question\": question, \"context\": context})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03e6ea25-812c-43e7-906e-367429b3e7d1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
